Statistical Methods in Measuring Search Engine Performance

There is a growing need for a set of benchmarking tests for measuring search engines, because research on search engine evaluation is inconsistent in methods and approaches. Using these tests researchers can make comparisons between different search engines and carry out longitudinal studies on a particular search engine with a degree of consistency. Several problems emerge during the evaluations of search engines which are needed to be solved to measure search engine performance objectively. This paper presents a brief review of the different statistical methods and approaches utilized in search engine evaluations. It describes six significant research projects with their fulfilled activities and statistical findings. It also serves as a guideline for choosing an appropriate statistical method for use in search engine evaluations. 1. General problems related to search engine evaluations Before discussing issues connected to search engine evaluation, a clear definition of search engine will be given. The term search engine refers to a general class of programs that enable users to search for documents on the World Wide Web. These programs index documents, then attempt to match documents relevant to a user’s search requests. In evaluations researchers measure the quality of Web search services which can be determined on the basis of several measures. However, it is extremely difficult to find reliable measures that reflect this quality appropriately. Oppenheim [8] suggested that benchmarking tests should include the following measures at the very minimum: 1. precision; 2. relative recall; 3. speed of response; 4. consistency of results over an extended period; 5. proportion of dead or out of date links; 6. proportion of duplicate hits; 7. overall quality of results as rated by users; 8. evaluation of GUI for user friendliness; 9. helpfulness of help and variations of software for new and experienced users; 10. options for display of results; 11. presence of adverts; 12. coverage; 13. estimated/expected search length; 14. length and readability of abstracts; 15. search engine effectiveness. 2000 Mathematics Subject Classification. 62P30, 68M20.